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The  Rise  of  Blue  Collar  Processing 
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In  DARPA’s  lifetime  the  user/cpu  ratio  has  inverted 
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DARPA  Director  Lauds  Winning  Embedded 
Processing  Solution  to  DARPA  Grand  Challenge 
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Photo  courtesy  of  DARPA 


GCi:  Embedded  DSP  Algorithms  Mapping  to 
non-traditional  commodity  architectures 


Grand  Challenges  and  Emerging  Perspectives : 

A  few  examples 


GC2:  Co-Design  of  Algorithms  and  Architecture 
Towards  Commodity  Nonlinear  DSP 


GC3:  Integrated  Sensing  with  Processing 

-  System  level  Co-design:  Sensor/Processor/Algorithms 

-  “compressed”  sensing/processing 


Low  cost,  low  power 
NonLinear  EQ  for  high 
system  dynamic  range 


Current  airborne  Advanced  EO/IR  System 


l  m  resol’n  over  lkm2 
0.25  Mpixel  @  10  Hz 


•  real-time  4  PPC  7410  500 
MHz  processors,  close  to 
4.93  GFLOPS 


PROCESSING  GRAND  CHALLENGE  l 
EO/IR  Space  Time  Adaptive  Processing  OSTAP 


IR-STAP  track  before  detect 
Small  targets  in  clutter 


•  512  X  512  IR  images  at  2 
Hz  rate,  36  target  velocity 
hypotheses 

To  move  beyond  TIVO  in  the  sky: 

Need  TeraFlops  Scale  Embedded  Processor 

25,600  windows  Upcoming  Systems 

5x5x5  matched  filter  #  100  Mpixels  @  2  Hz 

10  target  models 
10  second  update 
10,000  targets 


A  Different  World  in  Embedded  Processing 


•Increased  capability,  programmability 
•Towards  portable  form  factor,  power 


Open  standards  for  rapid  software  development 
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DoD  Sensor  processing  challenges 
• Complex  geometric  inverse  problems 
•Form  Factor  Limitations  $3.0 
•Curse  of  the  small  market 
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•Hardware  $$ 
•Software  $$$$ 


Estimated  DoD  Expenditure  ($B) 
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Phase  I  in  Perspective: 

STAP-BOY  in  “The  Embedded  Sensor  Processor  Trade  Space 
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What  if  we  do  these  the 
usual  way  rather  than 
using  STAP-BOY? 
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Phase  I  in  Perspective: 

STAP-BOY  in  “The  Embedded  Sensor  Processor  Trade  Space” 


What  if  we  do  these  the 
usual  way  rather  than 
using  STAP-BOY? 


Page  9 


1000++ 


Single  Chip  meets 


Global  Hawk  throughput 
reqm’nts 


STAPBOY  Oi 
6oW  .3k$ 


Phase  I  in  Perspective: 

STAP-BOY  in  “The  Embedded  Sensor  Processor  Trade  Space” 


>12  chips  -1.5  kW,  iok$ 
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What  if  we  do  these  the 
usual  way  rather  than 
using  STAP-BOY? 


EO/IR  Track-before-detect  (dim 
targets)  Change  detection 
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1000++ 


STAPBOY  02  :  ScalableHW/SW 
Dual  mobile  GPU  w/  route/format  fabric 
.5  TFLOP  box:  100-200W  (4  brds  )  ~iok$ 


RISC/DSP 


Constant 


67Mpixel 


IOQ 
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Phase  2:  Can  we  scale  the  STAP-BOY  low  cost  low  power 
hardware  and  software  to  required  system  performance? 


AMD  server 
120  W  ~.8k$ 


EO/IR  Track-before-detect  (dim 
targets)  Change  detection 
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Next  Challenge: 

Advancing  Sensor  Resolution  Drives  Processing  Well  Beyond  CPU/DSP 
and  ASIC  Envelopes  for  Cost  &  Comms  Constrained  Platforms 


267Mpixel 
4  km2  Qq2 
1  ft.  res.L^^ 
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PROCESSING  GRAND  CHALLENGE  2 

Getting  a  clear  view  through  the  RF  front  end 


Complex  RF  environment  Distorted  digital  Representation 


Rapid  Proliferation  of  LPI:  UWB,  spread,  hopped,  coded... 
Digital  Receivers  not  keeping  up 
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Attacking  this  problem 
through  a  revolution  in 
nonlinear  digital  signal 
processing 


10  10' 

Effective  BW  MHz 
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Co-Designed  Nonlinear  Equalization  (NI 
Advanced  Signal  Processing  for  Enhanced  Dynam 
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Representative  System 


From 

Antennas 


Digital  Receiver  N 
Digital  Receiver  1 


Program  Goal 
+  25  dB  IFDR 
@50oMHz  IBW 
<  5  Watts 
in  real  time 


D  A  A 
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M/U 
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Sources  of 
Nonlinearities 


20  Year 
leap! 


Effective  Sampling  Rate  =  2x  BW  (MHz) 


Before  Equalization 
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Objective:  Overcome  sensor  device  performance  and  cost  limitations  by 

VLSI  implementation  of  advanced  nonlinear  algorithms 

Key  Issues:  The  Curse  of  Dimensionality 

•Computational  Burden  of  NLEQ 
•Training  Cost  of  NLEQ 

Approach:  Compressed  Representations.  Algorithm/Hardware  co-design 


Input  Excitation 


f! 


Frequency 


Nonlinear  Representations  for  RF  chain 
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Wideband  Digital 
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Equalizer 
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Two-Tone 

Input  Response  Example 


Example  Equalizer 
Architecture 

— I  y(n) 


Static  Adaptive 
Nonlinear  linear 


P  PNN  \ 

y(n)  =  YjyP(n)=Hll'"llCp\nv-’np)x{n-n\)---x\n-np) 


p- 0 


p=0  n{=0  n 


-  Generalization  of  Taylor  series  (N=o) 

-  Generalization  of  linear  FIR  filter  (P=i) 

-  Naively,  size  scales  as  Np  for  memory  N  and  nonlinear  order  P 

-  Training  NLEQ  looks  like  a  high-d  optimization  problem 
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Co-Design  of  algorithm  and  hardware: 
Low-power  VLSI  NLEQ  Implementation 


NonLinear  Equalizer 
Algorithm  / Architecture* 


PHoCS  Compressed  Nonlinear 
representation 

Distributed  Polyphase  Block- 
Floating-Point  Residue 
Arithmetic  Architecture 

500  -  2000  OPS/Sample 

25,000  - 100,000  Low-Power 
Multipliers 

20,000  -  80,000  Accumulators 


IC  Architecture  and 
Floor  Plan* 


Clocks 
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Memory 

Control 

Systolic  architecture 
minimizes  high-speed  comm, 
path  lengths 

High  clock  rate  processing 
across  entire  die  area 

Two  Input  Ports,  Two  Output 
750  MSPS  Demultiplexed 
1500  MSPS  Full  Rate 

Control 


Processor  Die 


0.09-0.13  Micron  CMOS 
30M  -  100M  Devices 
2-4  cm2  Die  Area 

Low-Power  Low-Vt  Dynamic 
Logic 

~iTOPS,  <3  Watts 

Low-power  LVDS  compatible 
I/O  drivers 
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Making  NLEQ  a  reality  is  a  DARPA  style  Challenge 


Pentium  4  die 


•  1300  dies  required  for  l  Teraops 
NLEQ 

- 150  KW  total  power 

•  Assumptions  for  each  die 

-3.8  GHz 
- 115  Watts 
- 10%  code  efficiency 


Phase  2  NLEQ  DSP  Die 


•  1  die  required  for  l  Teraops 
NLEQ 

-<5  W  total  power 

•50,ooox  power  efficiency 

•  i,300x  computational 
throughput  density 

•65,ooo,ooox  total  figure  of 
merit  of  computational  density 
times  power  efficiency 
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S-Band  Receiver  and  ADC  (IM/IHz  Bandwidth) 


LNA  design  optimize 
for  low  noise  figure 


Current  circuit  designs  can  be 
costly  when  optimizing  both 
low  noise  and  high  dynamic 
range 
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10  MHz  Bandwidth:  ADC  Unequalized 
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Embedded  Non  Linear  Signal  Processing  simplifies  circuit  designs  if 
used  early  in  the  design  process 


Brains  and  bodies  evolved  together 


Embedded  processing  is  optimized  for  the  needs  of  the  system,  say  a  sensor 
system  like  a  camera.  Why  can’t  the  camera  be  optimized  for  the  processing? 


the  film,  which  makes  about  as  much  sense  as  automobiles  circa  1910,  which  were 

horse-drawn  carriages  without  the  horse.  A  car  owner  of  that  time  would  be 
pretty  shocked  by  what's  in  a  showroom  now.  Camera  stores  of  the  future  will 


surprise  US  just  as  much ”  “Nathan  Myhrvold,  former  CTO  of  Microsoft,  co-founder  of 
Intellectual  Ventures,  NY  Times,  5  June  2006 


Structural  &  functional  impact  of  co-design  over  the  entire  system? 
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Co-Design  opens  a  new  design  space 

between  Camera  and  Computed  Imaging 


Image 
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‘Load  Balancing”  between  Photon  Processing 

and  Bit  Processing 


Image 

Image 

Image 
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‘Load  Balancing”  between  Photon  Processing 

and  Bit  Processing 
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Radically  transform  form,  fit  and  function  of  imaging  sensors 
Integrated  systems  for  transforming  photons  to  information 
Simplified  Manufacturing  and  Integration  Process 


APPROACH: 


I _  _  |  t 

Propagation  from  pupil  to  detector 
Inverts  Propagation  from  object  to  pupil 


nrtrol 


•Joint  optimization  of  optics,  sensors,  post-processing  algorithms 
•Unconventional  wavefront  mapping  (well  conditioned  encoding) 
•Information-rich  parameters  for  (compressive)  measurement 
•Integrated  and  simplified  manufacturing  approaches 


Page  22 


Folded  Wavefront  Coded  Telephoto 


*2  -i 

2  =  (»■:= 

\  ■  r  in  =  i 


2zma:!! 

4§  m  v5u 

•*r  ms  i 


5li  ... 

61*  HI 


-2  -i 

2  =  ma'&r 

3  =  III  ™ li 

4  =  111  :b« -Ml 

elm  111=  1 


2.2m  In  focus  (2.6  m)  3m 


Page  23 


mr 

L.  ■*  i 


Montage  Nano  vs  Canon  SD30:  Resolution  Results 


Montage  MDO  “Nano” 

2  Megapixel,  ultra-compact  camera  30g 
(<  15  g  w/o  metal  housing) 


<  20mm2 


-30  lp/mm 


6.3  -  14.9mm 


Multi-element 

refractor 


80mm2 


100  lp/mm 


43mm 


Single  element 
folded  reflector 


-30mm 


5mm 


Canon  Powershot  SD30 

5  Megapixel  compact  camera,  i20g 


Light 

collection 


Resolution 
(2.2m  range) 

Focal  length 


Optics 


Optical  track 
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Montage  Nano 
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0  Canon  SD30  Compact  Camera 
■  Montage  Nano 
▲  Phase  1  Montage  8-Fold 
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Lenslet  Array  LWIR  camera  vs.  SOA 
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MONTAGE 

tiki 


How  is  it  done? 


1)  Forward  Camera  Model 
Predicts  camera  data  D  produced  by  a  given  scene 

2)  Solve  Imaging  Inverse  Problem 
Given  data,  estimates  scene  that  produced  it 

=  argminjz  (s;D  )+P  (y)} 

seS  K  ' 


S 


Discrepency  between  observed  Data  D  &  Penalty  for  unlikely  or 
Data  the  Camera  Model  predicts  from  s  unphysical  choices  of  s 

Innovations:  COMPRESSIVE  ENCODING 
CO-DESIGN  OF  CAMERA  &  INVERSE  ALGORITHM 
SPARSITY  AS  CONSTRAINT 


Sparsity  Regularized 

-w  • 


Reconstructed  Image 
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Exploiting  Sparsity  in  Compressed 
Sensing/Processing 


u  Compress:  Compute  hard,  throw 
away  80%  of  the  Megapixels  for 
Downlink  through  tiny  pipe 


High  Resolution  I  nage  : 

Big  Optics,  Megapixel  Data 

Uncomfortable  Observation: 


Compressed  Camera  directly  measures 
Compressed  representation  for  downlink 


Compressibility  means  most  of  the  data  in  the  megapixels  is  useless 

BUT 

The  gold  is  hidden  in  a  broad  spatial  bandwidth 

Shannon:  must  measure  the  megapixels  to  capture  the  hidden  gold 
then  sieve  it  out  with  digital  processing.  Pessimistic! 

Compressive  Sensing  (Analog-to-lnformation): 

Nonadaptive  Measurement  at  20%  of  Megapixel  rate  gets  the  gold 
Simpler  Sensor.  No  Digital  Compression  on  the  platform. 
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Power  to  the  Processors  (right  on!) 


Photo  courtesy  of  DARPA 


Research  Labora 
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Because  of  this,  the  Sensor  System  Physical  Layer.. 


Raw  sensor  data: 

Huge  Volume 
low  information/sample 

Curse  of  Dimensionality 


quantization  error 
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Physical  Field  Digital  Representation  Transformed 

(continuum)  (finite  precision  finite  dimensional)  Digital  Representation 


PHYSICAL 

LAYER 


EXPLOIT  A  TIO. 


Raw  sensor  data:  Enhanced  feature  data: 

low  information/dimension  more  information/dimensioni 


Sensor:  Work  hard  in  the  physical  layer  to  measure  everything  at  the  highest  fidelity 
Processor:  Work  hard  to  throw  away  most  of  that  data  to  get  at  the  “good  stuff” 


Growing  Decomposition  of  Sensing/Processing  Systems 


Field 

Stop 


Focal 

Plane 

Array 


Cooley  Tukey  FFT 
60  65  Subspace  Methods 


Wavelets 


ApproximationOO 
Optimization  i 

—  I  ► 


Best  Basis 


• Increased  complexity ,  dimensionality  of  sensing  and  exploitation  tasks 
• Decoupling  of  Digital  and  Analog  subsystems ,  independently  designed  and  optimized 
• Steady  Moore’s  Law  growth  of  digital  throughput  (integration  growth  path) 

• Equal  impact  of  fast  algorithms  (representations),  but  sporadic,  unpredictable  page  32 


Mega  Pixel  Trends:  Keep  on  keeping  on? 


JjItnriLl  Ti'smafoi'irifiLl  Synib© 

u  iiiiij  (finite  precision  iSnlte  dimensional)  Digital  Representation 


Physical  Field 


PHYSICAL 
LAYER 


1 


^ r 


/d igitize  ^wrything  at  hfi 


•  A  No-brainer 

•  No-brainer!  Maybe  not  always  true! 

-  Slow,  expensive  growth  rate  of  resolution  in  A/D’s 

-  Curse  of  dimensionality:  shouldn’t  do  it  even  if  you  could! 

•  What  might  be  better? 

-  Dedicate  Front  end  resources  to  measuring  more  of  the  “good  stuff’ 

-  Sensing  “features”  at  front  end  could  reduce  load  on  A/D  and  subsequent  digital 
processing  and  exploitation. 

-  If  done  properly,  overall  system  performance  can  be  improved  even  with  the  lower 
requirements  on  subsequent  digital  processing 


•  Problem:  the  “smarts”  for  finding  the  “good  stuff”  are  behind  the  ADC! 
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ISP:  A  Modest  Proposal  in  1999 


Today’s  Sensor  Systems  are  typically  feed-forward  networks  for 
transforming  information  in  specialized  stages. 

Dimension  reduction  is  mainly  applied  at  the  back  of  the  bus 


Physical  Field 
(continuum) 


Digital  Representation 
(finite  precision 
finite  dimensional) 


Transformed 
Digital  Representation 


Symbolic  Output 


PHYSICAL 

E  'HON 

LAYER 

Raw  sensor  data: 
low  information/ dimension 


DSP  produces  data  reps  with 
higher  information/ dimension 


Fast  approximate 
solution  of  a  high 
dimensional  non 
linear  optimization 
problem 
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Meeting  the  Challenges:  Integrated  Optimization  of  the  System 


Joint  Optimization  of  sensor  system  as 
network  of  overlapping  functionalities 


Components  will  have  overlapping  and  dynamically  reconfigurable  roles  and 
full  network  connectivity.  (LOAD  BALANCING^ 

Make  “customized”  measurements  at  physical  layer  under  real-time  feedback 
control  from  the  exploitation  and  processing  system.  “20  Questions” 

Manage/Prioritize  data  stream  to  affordable  levels  without  discarding  needed 

information 
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If  that’s  not  bad  enough,  we  are  building  an 
internet  of  the  damn  things.... 
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Uncomfortable  Observation: 

Compressibility  means  most  of  the  megapixels  are  useless 


Vision: 

Compressed  Camera  i  '  sofy  measures 
Compressed  representation  for  downlink 


Reality: 

High  Resolution  Image  : 

Big  Optics,  Megapixel  Data 


Sensor  Technology:  Too  much  of  a  good  thing? 


Compress:  Compute  hard,  throw 
away  80%  of  the  Megapixels  for 
Downlink  through  tiny  pipe 


BUT 


The  gold  is  hidden  in  a  broad  spatial  bandwidth,  need  smarts  to  find  it! 


Shannon:  must  measure  the  megapixels  then  sieve  the  hidden  gold 

Perhaps  Shannon  had  good  reason  to  look  worried? 
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Sensor  Trend:  Measure/digitize  everything  possible 
“  '  Mega  pixels  =  goodness! 


Raw  sensor  data: 

Huge  Volume 
low  information/sample 

Curse  of  Dimensionality 


quantization  error 
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Bellman  and  th 


Beware  The  Curse  of  Dimensionality 


-  Concept  named  by  Richard  Bellman 


-  Extraordinary  growth  in  difficulty  of  doing  business  in  higher 
dimensional  spaces 


•  Optimization,  Approximation 

-  Eg:  “reasonable”  function  of  d  variables  xx  ,x2, ...  xd  o<xi<i 

-  0(i/s)d  samples  to  obtain  0(s )  accuracy 


Would  need  100  sample  points  in  the  2-d  domain 


Statistics  Version:  Requirement  on  priors/training  to  do  meaningful 
inference 
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m.  w 

Ain’t  always  the  Place  to  Be 

ensions  are  strange 


The  odd  geometry  of  high 
Wasteful  packaging:  The  box  for  the  high-d  golf  ball  is  essentially  empty! 
Volume  is  in  the 


corners 


volume  of  the  hypercube  is  (2r)‘ 


volume  of  the  sphere  is 


2rair^9 

dr(d/2) ■ 


n 


d/2 


d2d~1r(d/2) 


Most  of  uniform  samples 
are  in  the  corners 


•Makes  inference  unreliable  w/o  more  prior 
or  training  data 

•Tends  to  make  computation  intractable  increasing 

lest  set  error 

•If  important  structure  is  really  low-d,  most 
of  the  high-d  features  mainly  drive  noise  into 
the  application 


Dimensionality 

Required  Sample  Size 

1 

4 

2 

19 

5 

786 

7 

10,700 

10 

842,000 

Required  samples  to  estimate 
Gaussian  density  accurately  at  center 


increasing  -feature  size 


Bayes  error 
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Because  of  this,  the  Sensor  System  Physical  Layer.. 


Raw  sensor  data: 

Huge  Volume 
low  information/sample 

Curse  of  Dimensionality 


quantization  error 
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Physical  Field  Digital  Representation  Transformed 

(continuum)  (finite  precision  finite  dimensional)  Digital  Representation 


PHYSICAL 

LAYER 


EXPLOIT  A  TIO. 


Raw  sensor  data:  Enhanced  feature  data: 

low  information/dimension  more  information/dimensioni 


Sensor:  Work  hard  in  the  physical  layer  to  measure  everything  at  the  highest  fidelity 
Processor:  Work  hard  to  throw  away  most  of  that  data  to  get  at  the  “good  stuff” 


Growing  Decomposition  of  Sensing/Processing  Systems 


Field 

Stop 


Focal 

Plane 

Array 


Cooley  Tukey  FFT 
60  65  Subspace  Methods 


Wavelets 


ApproximationOO 
Optimization  i 

—  I  ► 


Best  Basis 


• Increased  complexity ,  dimensionality  of  sensing  and  exploitation  tasks 
• Decoupling  of  Digital  and  Analog  subsystems ,  independently  designed  and  optimized 
• Steady  Moore’s  Law  growth  of  digital  throughput  (integration  growth  path) 

• Equal  impact  of  fast  algorithms  (representations),  but  sporadic,  unpredictable  page  43 


Mega  Pixel  Trends:  Keep  on  keeping  on? 


Digital  rl'nnniunn b d  Synib© 

infi  rm  n  rnj  (finite  precision  ifinite  dimensional)  'Digital  Representation 


Physical  Field 


PHYSICAL 
LAYER 


1 


^ r 


/d igitize  ^wrything  at  hfi 


•  A  No-brainer 

•  No-brainer!  Maybe  not  always  true! 

-  Slow,  expensive  growth  rate  of  resolution  in  A/D’s 

-  Curse  of  dimensionality:  shouldn’t  do  it  even  if  you  could! 

•  What  might  be  better? 

-  Dedicate  Front  end  resources  to  measuring  more  of  the  “good  stuff’ 

-  Sensing  “features”  at  front  end  could  reduce  load  on  A/D  and  subsequent  digital 
processing  and  exploitation. 

-  If  done  properly,  overall  system  performance  can  be  improved  even  with  the  lower 
requirements  on  subsequent  digital  processing 


•  Problem:  the  “smarts”  for  finding  the  “good  stuff”  are  behind  the  ADC! 
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ISP:  A  Modest  Proposal  in  1999 


Today’s  Sensor  Systems  are  typically  feed-forward  networks  for 
transforming  information  in  specialized  stages. 

Dimension  reduction  is  mainly  applied  at  the  back  of  the  bus 


Physical  Field 
(continuum) 


Digital  Representation 
(finite  precision 
finite  dimensional) 


Transformed 
Digital  Representation 


Symbolic  Output 


PHYSICAL 

E  'HON 

LAYER 

Raw  sensor  data: 
low  information/ dimension 


DSP  produces  data  reps  with 
higher  information/ dimension 


Fast  approximate 
solution  of  a  high 
dimensional  non 
linear  optimization 
problem 
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Meeting  the  Challenges:  Integrated  Optimization  of  the  System 


Joint  Optimization  of  sensor  system  as 
network  of  overlapping  functionalities 


Components  will  have  overlapping  and  dynamically  reconfigurable  roles  and 
full  network  connectivity.  (LOAD  BALANCING^ 

Make  “customized”  measurements  at  physical  layer  under  real-time  feedback 
control  from  the  exploitation  and  processing  system.  “20  Questions” 

Manage/Prioritize  data  stream  to  affordable  levels  without  discarding  needed 

information 
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